Predicting a light probe for an outdoor image

ABSTRACT

Methods and systems for predicting light probes for outdoor images are disclosed. A light probe database is created to learn a mapping from the outdoor image&#39;s features to predicted outdoor light probe illumination parameters. The database includes a plurality of images, image features for each of the plurality of images, and a captured light probe for each of the plurality of images. A light probe illumination model based on a sun model and sky model is fitted to the captured light probes. The light probe for the outdoor image may be predicted based on the database dataset and fitted light probe models.

TECHNICAL FIELD

The present disclosure relates generally to light probes, and moreparticularly, some embodiments relate to systems and methods forpredicting light probes for outdoor images.

DESCRIPTION OF THE RELATED ART

Image-based lighting technology is a key component of creatingphotorealistic computer-generated content. Movie special effects shots,for example, often contain several virtual objects that are inserted andrelighted into the movie after the movie was filmed. Current methods ofrealistically illuminating these special effects objects require directcapture of the illumination conditions of the scene in which the virtualobject is inserted. This requires careful planning, precise image datahandling techniques, and cannot be applied to photographs for sceneswhose illumination conditions were not captured. Application of suchphotorealistic lighting techniques is monetarily and temporallyexpensive. Accordingly, methods of accurately predicting the lightingconditions of an image are desirable.

BRIEF SUMMARY OF THE DISCLOSURE

According to various embodiments of the disclosed methods and systems,light probes are predicted for outdoor images based on a learned mappingfrom the outdoor image's features to light probe illuminationparameters. The outdoor image's features may include features computedfrom a surface normal. In one embodiment, the learned mapping is basedon a light probe database that includes a plurality of images, imagefeatures for each of the plurality of images, and a captured light probefor each of the plurality of images.

In one particular embodiment, the learned mapping is based on a sunmodel for sun parameters and a sky model for sky parameters. In thisembodiment, the light probe illumination parameters comprise the sunparameters and the sky parameters. The sun parameters may comprise sunposition, sun angular variation, and sun color. The sky parameters maycomprise sky color, sky angular variation, and sky turbidity.

In one embodiment, the sun model is based on a double-exponential modeland the sky model is based on a Preetham model. In other embodiments,the sun model may be based on a von-Mises Fisher model, and the skymodel may be based on a cosine model or a von-Mises Fisher model. In yetother embodiments, any combination of sun model and sky model may beused. The sun model and sky model are fitted to the light probe databasecaptured light probes to create fitted models.

In further embodiments, a light probe is predicted by predicting the sunposition based on a probabilistic model and predicting the skyparameters and other sun parameters using a regression technique. Thislight probe prediction process uses the fitted models. In yet furtherembodiments, a virtual object is inserted into the outdoor image andlighted using the predicted light probe.

Other features and aspects of the disclosed method and system willbecome apparent from the following detailed description, taken inconjunction with the accompanying drawings, which illustrate, by way ofexample, the features in accordance with embodiments of the disclosure.The summary is not intended to limit the scope of the claimeddisclosure, which is defined solely by the claims attached hereto.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The figures are provided for purposes of illustration only andmerely depict typical or example embodiments of the disclosure.

FIG. 1A illustrates a communications environment in accordance with thepresent disclosure.

FIG. 1B illustrates a high-level block diagram of a light probeprediction device.

FIG. 2A is an operational flow diagram illustrating an example processfor creating fitting models for light probe illumination parametersbased on a created light probe database. The fitted models may be usedin a light probe prediction process.

FIG. 2B is an operational flow diagram illustrating an example processfor creating a light probe database that may be used in the process ofFIG. 2A.

FIG. 3 illustrates an example set of light probe data for one scene thatmay be used in a light probe database.

FIG. 4 illustrates two example sun and sky model combinations that maybe used as a model for light probe illumination conditions for anoutdoor image.

FIG. 5 is an operational flow diagram illustrating an example processfor predicting a light probe for an outdoor image based on a learnedmapping from the outdoor image's features to light probe illuminationparameters.

FIG. 6 illustrates an example image that uses a predicted light probe tolight an inserted virtual object.

FIG. 7 illustrates an example computing module that may be used toimplement various features of the systems and methods disclosed herein.

DETAILED DESCRIPTION

FIG. 1A illustrates a communications environment for light probeprediction in accordance with the present disclosure. In communicationsenvironment 100, camera 110 captures a photograph of a scene that isthen transmitted to light probe prediction device 130 over communicationmedium 120. Light probe prediction device 130 may comprise any computingdevice (tablets, PDA's, smartphones, cellphones, palmtops, laptops,etc.), workstations or servers, or any other type of general-purposecomputing device configured to receive captured photographs and predictthe light probe for the scene (i.e. the scene's illuminationsconditions) based on the captured photograph's properties. Light probeprediction device 130 may use the predicted light probe to relight avirtual object (e.g. a character, an animal, a building, a tree, etc.)into the photographed scene. In one embodiment, camera 110 may comprisea video recording device that captures a plurality of images (frames).In this embodiment, light probe prediction device 130 may predict alight probe for each of the plurality of frames to relight a virtualobject into the video.

Communication medium 120 may comprise a communications network such as acellular telephone network, a local area network (LAN), a wireless LAN(WLAN), a wide area network (WAN), a personal area network (PAN), aportion of the Internet, or any combination thereof. The medium 120 maybe a wireless network system such as a cellular network, a wirelesspersonal area network, a wireless local area network, a Bluetoothsystem, or other similar communication medium. The medium alternativelymay be a wired system, such as a coaxial cable system, a fiber opticcable system, an Ethernet cable system, a USB system, or other similarcommunication medium.

Light probe prediction device 130 may be implemented in any environmentwhere the realistic relighting of virtual objects into outdoorphotographs is desired. In one embodiment, the environment does notrequire real-time light probe prediction. For example, light probeprediction device 130 may be used in a movie post-filming process torelight virtual objects such as special effects or CGI characters intofilmed movie frames. As another example, light probe prediction device130 may be used in the design of video games to dynamically insertvirtual video game characters or objects into photographs or video. Inanother embodiment, the environment is a dynamic environment thatrequires immediate insertion of a virtual object into images. Forexample, the environment may comprise a theme park where camera 110photographs theme park attendees and light probe prediction device 130immediately thereafter inserts and relights a virtual object into thephotograph. In this embodiment, camera 110 and light probe device 130may be integrated into one module.

FIG. 1B is a high-level block diagram of a light probe prediction device130 that may be used in accordance with the described system. Lightprobe prediction device 130 may comprise a storage 131 configured tostore a captured light probe database 132, processor 133, light probeprediction application 134, display 135, and connectivity interface 136for connecting to camera 110 through communication medium 120.

Display 135 may be configured to display a photograph or video that wasrelighted with a virtual object using light probe prediction application134. Alternatively, display 134 may be configured as a peripheral for anoperator of light probe prediction device 130 to interact with lightprobe prediction application 134 through a user interface (not shown).

Light probe prediction device 130 implements a light probe predictionapplication 134 for predicting the light probe of an outdoor image orphotograph for subsequently inserting a virtual object into the imageand relighting it. Light probe prediction application 134 may beintegrated as part of an animation application, a movie editingapplication, an image editing application, a video game designapplication, or some combination thereof. Light probe predictionapplication is used in conjunction with light probe database 132(described below) to provide a system and method for predicting thelighting conditions of an outdoor image so that a virtual object may beinserted and relighted in the image. As illustrated in FIG. 1B, Lightprobe database 132 is integrated with light probe prediction device 130.In another embodiment, light probe database 132 is accessible by lightprobe prediction device 130 over a communications interface.

FIG. 2A is an operational flow diagram illustrating an example processfor creating fitting models for light probe illumination parametersbased on a created light probe database. The fitted models maysubsequently be used in a mapping for predicting an image's light probebased on the image's features. At operation 210, a light probe databaseis created. FIG. 2B is an operational flow diagram illustrating anexample process 210 for creating a light probe database that may be usedin process 200. Process 210 creates a light probe database 132 that maybe used in a light probe prediction process. As will be furtherdescribed below, the light probe database comprises a plurality ofoutdoor photographs or images, image features for each of the pluralityof photographs, and a plurality of captured light probes associated withthe plurality of photographs. The photographed locations are capturedunder a plurality of illumination conditions.

With reference now to process 210, at operation 211 a camera captures alocation image for an outdoor scene. The image may be captured atapproximately eye level or another perspective that facilitates relativealignment of the image and associated light probe. A single operation211 may comprise taking one image or a plurality of images for theoutdoor scene at a particular time. The captured outdoor scene may beselected based on desired parameters that improve the accuracy of themapping. In one embodiment, for example, the desired parameters comprisethe presence of man-made structures (e.g. buildings) with relativelyplanar surfaces and relatively few non-lambertian structures. In anotherembodiment, the desired parameters may vary based on the expectedparameters of the photograph with unobserved illumination conditions forwhich a light probe is predicted. In another embodiment, the desiredparameters may vary based on the desired scope of applicability of themapping (e.g. how much it can be generalized while maintaining athreshold predictive accuracy).

At operation 212, a ground truth light probe is captured for one or morelocation images. In one particular implementation, the light probe iscaptured by pointing a camera at the sky in the direction opposite thecaptured scene and taking one or more photographs (e.g. 9 exposures inburst mode). In an alternative implementation, a camera may be installedat a high vantage point (e.g. a roof of a nearby tall building), andmultiple exposures of the light probe may be captured. If multipleexposure images are captured, they may be combined to create a singlehigh-dynamic-range (HDR) representation of the light probe. The cameraused to capture the light probe may comprise a wide-angle fisheye lensor other lens configured to take wide-angle photographs. In oneembodiment, the camera lens is calibrated such that the light probe canbe mapped to an angular environment map representation. If more imagesare required for the light probe database (decision 213), operations 211and 212 are repeated. The desired size of the light probe database maydepend on the number of different scenes captured, the number ofdifferent illumination conditions captured, and the desired accuracy ofthe mapping.

In exemplary method 210, operation 212 is performed immediately afteroperation 211 to associate the captured light probe to the capturedlocation image. In some embodiments, a plurality of captured locationimages may be associated with one captured light probe. Each operationmay be performed with the same camera or a different camera. If theoperations are performed with different cameras, the camera clocks andwhite balance settings may be synchronized prior to beginning process210.

At operation 214, the captured location images and light probe imagesare calibrated and aligned. For example, in one embodiment the lightprobes are aligned by aligning the exposures of the skies, correctingfor camera displacements, and rotating the light probe to match the sunposition and orientation. In another embodiment, all of the capturedlocation images and light probes are intensity-normalized on the samerelative exposure based on their exposure value.

Image features are then recovered for each location image at operation215. Image features may comprise any features that may be used todetermine outdoor lighting conditions and direction. For example, imagefeatures may comprise the brightest and darkest regions of a structurein the image, shadows and their relative depth, and surface normals of astructure in the image. In one embodiment, recovery of image featuresfor each location image may comprise recovering a normal map for eachlocation image. In one embodiment, a normal map is recovered byreconstructing the image in three dimensions and aligning the image withthe camera. In one embodiment, the location image is reconstructed inthree dimensions using structure from motion. In this embodiment, onlythe objects of interest (e.g. man-made lambertian structures) aremodeled in three dimensions and other potential blockers that might castshadows on the object of interest are ignored. In an alternativeembodiment, potential blockers may be considered. In an alternativeembodiment, an existing manmade three-dimensional model of the object ofinterest is aligned to the image data. In this embodiment, a normal mapis recovered by back projecting each surface normal of the threedimensional model into the image. In one embodiment, recovery of thenormal map further comprises aligning the reference frame of the objectof interest with the world reference frame.

After completion of process 210, a light probe database dataset has beengathered. The dataset comprises images, the light probes associated withthe images, and the image features of the images. FIG. 3 illustrates anexemplary subset of data for a light probe database. The exemplary datacomprises three images 310 for a location, the normal maps 311associated with each of the images, and the light probes 312 associatedwith each of the images. The illustrated images are geometrically andradiometrically calibrated. With the light probe database created, lightprobe illumination models may be selected for modeling the illuminationparameters of a predicted light probe.

Referring now back to process 200, at operation 220 models are selectedfor modeling the illumination parameters of a light probe. In exemplaryprocess 200, the models are based on a sun model and a sky model fordescribing the outdoor illumination parameters. The selected sun and skymodels provide the benefit of providing low-dimensional models thataccurately capture the main sources of light variations in the sky withfew parameters. This saves computational resources by simplifying thetask of predicting light probes to predicting the values of the modelparameters.

In alternative embodiments, the models may be based on other outdoorlight sources such as, for example, a moon model or an artificial lightmodel. In these alternative embodiments, the light probe database maycomprise a different set of images (e.g. nighttime images with themoon).

In the illustrated embodiment, the sun and sky are modeled separately.In alternative embodiments, the sun and sky are modeled together. Withreference now to the sun model, the sun model comprises a plurality ofillumination parameters for modeling the lighting of the sun. Forexample, the sun model may comprise the illumination parameters of sunposition or direction, sun angular variation or spatial variation,and/or sun color. Each illumination parameter comprises one or moredimensions. For example, sun color may be modeled in three dimensionsusing an RGB color representation. In one embodiment, the sun may bemodeled with a von-Mises Fisher (vMF) distribution as described in A.Banerjee, I. S. Dhillon, J. Ghosh, and S. Sra. Clustering on the unithypersphere using von mises-fisher distributions. Journal of MachineLearning Research, 6:1345-1382, 2005. In this embodiment, the vMFdistribution is used to model the intensity of the sun's light directionin an RGB color channel as a function of the direction, color, anddiffuseness of the sun. In an alternative embodiment, the sun may bemodeled with a double-exponential distribution. In one examplemathematical implementation of the double-exponential distribution, thesun model may be defined by Equation (1):

f _(sun) =e ^(ge) ^(−k/γ)   (1)

Where g controls the width of the curve for low intensities (i.e. theshape of scattering close to the sun), k controls the width of the curveat high intensities (i.e. the shape of the sun), and γ is the angulardifference between the sun center and the light direction. In otherembodiments, other sun models may be used.

The sky model similarly comprises a plurality of illumination parametersof one or more dimensions for modeling the lighting of the sky. Forexample, the sky model may comprise the illumination parameters of skycolor, sky angular variation, sky turbidity, and/or the zenith angle oflight direction. In one embodiment, the sky may be modeled using the vMFdistribution in a similar manner to that described above. In analternative embodiment, the sky may be modeled using a Preetham model asdescribed in A. J. Preetham, P. Shirley, and B. Smits. A practicalanalytic model 842 for daylight. In SIGGRAPH, August 1999. In yetanother embodiment, the sky may be modeled using the classic non-uniformsky model (cosine model) described in P. Moon and D. E. Spencer.Illumination from a non-uniform sky. 826 Transactions of theIllumination Engineering Society, 37, 1942, which may be used torepresent the sky variation using a weighted cosine term. In otherembodiments, other sky models may be used.

The selected sun and sky models may be combined in any arrangement. Theselection of the models may depend on the expected image andillumination conditions of the photograph for which a light probe isbeing predicted (e.g. overcast or clear skies) and/or the desiredpredictive accuracy of the model (i.e. “fit”) versus computationalcomplexity. FIG. 4 illustrates two example illumination models that usedifferent sun and sky models. Illumination model 1 has 12 dimensions: 2for sun position, 1 for sun concentration, 3 for sun color, 3 for skycolor, and 3 for the sky costine term. Model 1 uses a vMF distributionfor modeling the sun and a cosine model for modeling the sky.Illumination model 2 has 11 dimensions: 2 for the sun position, 2 forthe sun concentration, 3 for the sun color, 3 for the sky color, and 1for the sky turbidity. Model 2 uses a double-exponential distributionfor modeling the sun and a Preetham model for modeling the sky.

Following the selection of a sun and sky model for modeling theillumination parameters of a light probe, at operation 230 the selectedsun model and sky model are fitted to the light probe database groundtruth light probes. In one embodiment, the sun and sky models may befitted to the ground truth light probes by applying a non-linearleast-squares fitting method. In this embodiment, the fitting method mayfurther comprise fitting parameters to ranges that avoid ambiguities inexplaining the causes of a sky's appearance. For example, the parameterfitting range may be chosen such that the fitted model always predictsthe most probable cause (i.e. composition of sun and sky) of particularillumination conditions. Alternatively, the fitted model may be chosenbased on the expected cause of the illumination conditions of theoutdoor images for which the light probes are predicted. With the lightprobe database and fitted models, a light probe may then be predictedfor an outdoor image of a scene.

FIG. 5 is an operational flow diagram illustrating an example process500 for predicting a light probe for an outdoor image 502 (e.g.photograph or video frames) based on a learned mapping from the outdoorimage's features to light probe illumination parameters. In exemplaryprocess 500, the lighting conditions for image 502 have not beendirectly captured. As further described below, process 500 predicts thelight probe for image 502 using only the image features of image 502,for example, a normal map 501 of image 502. The light probe illuminationparameters may be predicted independently of each other, in a pluralityof groups, or together in one group. The learned mapping relies onmodels fitted to a previously created light probe database 510, whichcomprises a plurality of images 511, image features 512 associated witheach of the plurality of images 511, and a plurality of calibrated lightprobes 513 associated with images 511. In some embodiments, images 511,image features 512, and calibrated light probes 513 may be distributedover multiple databases. Image features 512, may include, for example,brightest and darkest regions of a structure in the images, shadows andtheir relative depth, and surface normals of a structure in the image(e.g. recovered from a normal map of the image).

At operation 503, image features are extracted from image 502. In oneexemplary embodiment, the image features extracted from image 502 arethe same as the image features 512 previously extracted from images 511.Extracting the same features from image 502 enables a subsequentregression (described below) of those features onto database 510, wherethey were previously computed for other images 511. In one embodiment, anormal map 501 is extracted from image 502. Normal map 501 may comprisea plurality of surface normals of a structure or other object present inimage 502.

At operation 504, an illumination model with illumination parameters ispredicted for image 502 based on the extracted image features. Thepredicted illumination model relies on models previously fitted to lightprobe database 510. In one exemplary embodiment, the illumination modelis based on a sun model and a sky model respectively comprising sunparameters (e.g. sun position/direction, sun angular variation orspatial variation, sun color, etc.) and sky parameters (e.g. sky color,sky angular variation, sky turbidity, zenith angle of light direction,etc.)

In one embodiment, the sun direction is predicted independently of theother illumination parameters of the illumination model. In thisembodiment, the sun direction is predicted using a mapping from outdoorimage 502 and its normal map 501 to the sun direction. In a particularimplementation of this embodiment, it is assumed that each of image 502and a subset or all of images 511 comprise an object that is composed ofat least two approximately planar surfaces, each surface is mainlylambertian (e.g. non-lambertian materials such as windows, if any, aresparse), and the surfaces possess a similar distribution of materials.In this embodiment, the surfaces may or may not have a similardistribution of materials across different images. A probabilistic modelmay use the ratios of irradiance (intensity) of two planar surfaces withdifferent intensities to predict the most likely light direction givenimage 502 and its geometry. The probabilistic model samples the entirespace or a subspace of the possible sun positions based on fitted lightprobe database 510 and determines the probability that each sun positiongives the appearance of the scene based on the planar surfaces (i.e.surface normal). In one embodiment, the possible sun positions areweighted according to their likelihood given fitted light probe database510. The probabilistic model of this ratio and the fitted data of lightprobe database 510 are used to account for unknown variables. In analternative embodiment, the sun direction may be independently predictedfrom other illumination parameters using the known geometry of theoutdoor scene (e.g. ground plane and nearby occluders that generate castshadows).

In an alternative embodiment, the sun position may be predicted jointlywith the other illumination parameters. In this embodiment, the lightprobe database 510 is used to impose constraints on the illuminationparameters (i.e. determine what typical illumination parameters looklike). In one embodiment, joint prediction is performed using only theknown material properties (albedo) of the surfaces of the scene of image502. In an alternative embodiment, joint prediction is performed usingboth the known material properties of the surfaces of the scene, and theknown geometry of the outdoor scene of image 502 (e.g. ground plane andnearby occluders that generate cast shadows). The sun position may bepredicted relative to the camera or another reference point.

In one embodiment, illumination parameters other than sun position (e.g.sun angular variation, sun color, sky color, etc.) may be predictedusing a regression technique. In one embodiment, a kernel regressionfunction is used to learn a mapping from the image features of outdoorimage 502 to illumination parameters. Image features for the kernelregression may be selected based on their scene independence and abilityto retain important illumination information. The mapping parameters arelearned using the dataset of the light probe database 510. In onealternative embodiment, the sun position may also be predicted using akernel regression technique. In other embodiments, other regressionstechniques such as a linear regression may be used.

Once the illumination model with corresponding illumination parametersis predicted for image 502, at operation 505 a light probe is predictedfor image 502. Given a predicted illumination model with knownillumination parameters, the illumination color for any given lightdirection may be obtained by evaluating the model with those parametersat the given light direction. In one embodiment, the predicted lightprobe is generated by sampling a plurality of light directions spanningthe sky hemisphere, and evaluating the predicted illumination model ateach of the plurality of directions using the illumination parameters toobtain light colors for each of the plurality of directions.

Once a light probe is predicted for outdoor image 502, at operation 506a virtual object is inserted into the outdoor image and lighted usingthe predicted light probe. In one embodiment, the image is lighted usingthe image based lighting pipeline. In some embodiments, additionaltechniques may be used to model nearby geometry that may not beconsidered by the described method 500. FIG. 6 illustrates an exampleoutdoor image 600 with predicted light probe 605 and inserted andlighted virtual object 610.

In one embodiment, light probe prediction process 500 may be used todynamically model the lighting conditions of a video comprising aplurality of image frames. In this embodiment, the temporal movements ofthe image frame are incorporated in the light probe prediction model'sparameters. The predicted light probe may be temporally updated based ona predefined frequency, sudden changes in the lighting conditions of theimage scene, movements away from the image scene, the introduction ofobjects or occluders into the image scene, or some combination thereof.For example, in videos where the image frame remains relatively staticover time, the light probe predicted for one image frame may be reusedfor subsequent image frames, assuming the lighting conditions of thescene do not suddenly change. In one implementation of this embodiment,temporal smoothness (across frames) is imposed as a condition of themodel.

FIG. 7 illustrates an example computing module that may be used toimplement various features of the systems and methods disclosed herein.As used herein, the term module might describe a given unit offunctionality that can be performed in accordance with one or moreembodiments of the present application. As used herein, a module mightbe implemented utilizing any form of hardware, software, or acombination thereof. For example, one or more processors, controllers,ASICs, PLAs, PALs, CPLDs, FPGAs, logical components, software routinesor other mechanisms might be implemented to make up a module. Inimplementation, the various modules described herein might beimplemented as discrete modules or the functions and features describedcan be shared in part or in total among one or more modules. In otherwords, as would be apparent to one of ordinary skill in the art afterreading this description, the various features and functionalitydescribed herein may be implemented in any given application and can beimplemented in one or more separate or shared modules in variouscombinations and permutations. Even though various features or elementsof functionality may be individually described or claimed as separatemodules, one of ordinary skill in the art will understand that thesefeatures and functionality can be shared among one or more commonsoftware and hardware elements, and such description shall not requireor imply that separate hardware or software components are used toimplement such features or functionality.

Where components or modules of the application are implemented in wholeor in part using software, in one embodiment, these software elementscan be implemented to operate with a computing or processing modulecapable of carrying out the functionality described with respectthereto. One such example computing module is shown in FIG. 7. Variousembodiments are described in terms of this example-computing module 700.After reading this description, it will become apparent to a personskilled in the relevant art how to implement the application using othercomputing modules or architectures.

Referring now to FIG. 7, computing module 700 may represent, forexample, computing or processing capabilities found within desktop,laptop, notebook, and tablet computers; hand-held computing devices(tablets, PDA's, smart phones, cell phones, palmtops, etc.); mainframes,supercomputers, workstations or servers; or any other type ofspecial-purpose or general-purpose computing devices as may be desirableor appropriate for a given application or environment. Computing module700 might also represent computing capabilities embedded within orotherwise available to a given device. For example, a computing modulemight be found in other electronic devices such as, for example, digitalcameras, navigation systems, cellular telephones, portable computingdevices, modems, routers, WAPs, terminals and other electronic devicesthat might include some form of processing capability.

Computing module 700 might include, for example, one or more processors,controllers, control modules, or other processing devices, such as aprocessor 704. Processor 704 might be implemented using ageneral-purpose or special-purpose processing engine such as, forexample, a microprocessor, controller, or other control logic. In theillustrated example, processor 704 is connected to a bus 702, althoughany communication medium can be used to facilitate interaction withother components of computing module 700 or to communicate externally.

Computing module 700 might also include one or more memory modules,simply referred to herein as main memory 708. For example, preferablyrandom access memory (RAM) or other dynamic memory, might be used forstoring information and instructions to be executed by processor 704.Main memory 708 might also be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 704. Computing module 700 might likewise include aread only memory (“ROM”) or other static storage device coupled to bus702 for storing static information and instructions for processor 704.

The computing module 700 might also include one or more various forms ofinformation storage mechanism 710, which might include, for example, amedia drive 712 and a storage unit interface 720. The media drive 712might include a drive or other mechanism to support fixed or removablestorage media 714. For example, a hard disk drive, a solid state drive,a magnetic tape drive, an optical disk drive, a CD or DVD drive (R orRW), or other removable or fixed media drive might be provided.Accordingly, storage media 714 might include, for example, a hard disk,a solid state drive, magnetic tape, cartridge, optical disk, a CD orDVD, or other fixed or removable medium that is read by, written to oraccessed by media drive 712. As these examples illustrate, the storagemedia 714 can include a computer usable storage medium having storedtherein computer software or data.

In alternative embodiments, information storage mechanism 710 mightinclude other similar instrumentalities for allowing computer programsor other instructions or data to be loaded into computing module 700.Such instrumentalities might include, for example, a fixed or removablestorage unit 722 and an interface 720. Examples of such storage units722 and interfaces 720 can include a program cartridge and cartridgeinterface, a removable memory (for example, a flash memory or otherremovable memory module) and memory slot, a PCMCIA slot and card, andother fixed or removable storage units 722 and interfaces 720 that allowsoftware and data to be transferred from the storage unit 722 tocomputing module 700.

Computing module 700 might also include a communications interface 724.Communications interface 724 might be used to allow software and data tobe transferred between computing module 700 and external devices.Examples of communications interface 724 might include a modem orsoftmodem, a network interface (such as an Ethernet, network interfacecard, WiMedia, IEEE 802.XX or other interface), a communications port(such as for example, a USB port, IR port, RS232 port Bluetooth®interface, or other port), or other communications interface. Softwareand data transferred via communications interface 724 might typically becarried on signals, which can be electronic, electromagnetic (whichincludes optical) or other signals capable of being exchanged by a givencommunications interface 724. These signals might be provided tocommunications interface 724 via a channel 728. This channel 728 mightcarry signals and might be implemented using a wired or wirelesscommunication medium. Some examples of a channel might include a phoneline, a cellular link, an RF link, an optical link, a network interface,a local or wide area network, and other wired or wireless communicationschannels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to transitory ornon-transitory media such as, for example, memory 708, storage unit 720,media 714, and channel 728. These and other various forms of computerprogram media or computer usable media may be involved in carrying oneor more sequences of one or more instructions to a processing device forexecution. Such instructions embodied on the medium, are generallyreferred to as “computer program code” or a “computer program product”(which may be grouped in the form of computer programs or othergroupings). When executed, such instructions might enable the computingmodule 700 to perform features or functions of the present applicationas discussed herein.

Although described above in terms of various exemplary embodiments andimplementations, it should be understood that the various features,aspects and functionality described in one or more of the individualembodiments are not limited in their applicability to the particularembodiment with which they are described, but instead can be applied,alone or in various combinations, to one or more of the otherembodiments of the application, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus, the breadth and scope of the presentapplication should not be limited by any of the above-describedexemplary embodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, can be combined in asingle package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

While various embodiments of the present disclosure have been describedabove, it should be understood that they have been presented by way ofexample only, and not of limitation. Likewise, the various diagrams maydepict an example architectural or other configuration for thedisclosure, which is done to aid in understanding the features andfunctionality that can be included in the disclosure. The disclosure isnot restricted to the illustrated example architectures orconfigurations, but the desired features can be implemented using avariety of alternative architectures and configurations. Indeed, it willbe apparent to one of skill in the art how alternative functional,logical or physical partitioning and configurations can be implementedto implement the desired features of the present disclosure. Also, amultitude of different constituent module names other than thosedepicted herein can be applied to the various partitions. Additionally,with regard to flow diagrams, operational descriptions and methodclaims, the order in which the steps are presented herein shall notmandate that various embodiments be implemented to perform the recitedfunctionality in the same order unless the context dictates otherwise.

Although the disclosure is described above in terms of various exemplaryembodiments and implementations, it should be understood that thevarious features, aspects and functionality described in one or more ofthe individual embodiments are not limited in their applicability to theparticular embodiment with which they are described, but instead can beapplied, alone or in various combinations, to one or more of the otherembodiments of the disclosure, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus, the breadth and scope of the presentdisclosure should not be limited by any of the above-described exemplaryembodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, can be combined in asingle package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

What is claimed is:
 1. A computer-implemented method, comprisingpredicting a light probe for an outdoor image based on a learned mappingfrom the outdoor image's features to light probe illuminationparameters.
 2. The method of claim 1, wherein the learned mapping isbased on a light probe database comprising: a plurality of images; imagefeatures for each of the plurality of images; and a plurality ofcaptured light probes associated with the plurality of images.
 3. Themethod of claim 2, wherein the outdoor image's features comprisefeatures computed from a surface normal.
 4. The method of claim 3,wherein image features for each of the plurality of images in the lightprobe database comprise a normal map.
 5. The method of claim 2, whereinthe plurality of images comprise a plurality of locations captured undera plurality of illumination conditions.
 6. The method of claim 5,wherein the light probe illumination parameters comprise sun parametersand sky parameters.
 7. The method of claim 6, wherein the learnedmapping is based on a sun model for the sun parameters and a sky modelfor the sky parameters.
 8. The method of claim 7, wherein the sunparameters comprise sun position, sun angular variation, and sun color.9. The method of claim 8, wherein the sky parameters comprise sky colorand sky angular variation.
 10. The method of claim 7, wherein the sunmodel is based on a double exponential model.
 11. The method of claim 7,wherein the sky model is based on one of a Preetham model, a cosinemodel, or a von-Mises Fisher model.
 12. The method of claim 7, whereinthe sun model and the sky model are fitted to the light probe databasecaptured light probes to create fitted models.
 13. The method of claim7, further comprising inserting a virtual object into the outdoor imageand lighting the virtual object using the predicted light probe.
 14. Themethod of claim 9, wherein predicting a light probe for an outdoor imagecomprises: predicting the sun position based on a probabilistic model;and predicting the sun angular variation, sun colors, sky colors, andsky angular variation based on a regression technique.
 15. The method ofclaim 14, wherein the sun position is predicted relative to the camerathat took the outdoor image.
 16. The method of claim 14, wherein theregression technique is a kernel regression or a linear regression. 17.A system, comprising: a camera configured to take an outdoor image; anda computer configured to predict a light probe for the outdoor imagebased on a learned mapping from the outdoor image's features to lightprobe illumination parameters.
 18. The system of claim 17, wherein thelearned mapping is based on a light probe database comprising: aplurality of images; image features for each of the plurality of images;and a plurality of captured light probes associated with the pluralityof images.
 19. The system of claim 16, wherein the outdoor image'sfeatures comprises features computed from a surface normal.
 20. Thesystem of claim 19, wherein image features for each of the plurality ofimages in the light probe database comprise a normal map.
 21. The systemof claim 18, wherein the plurality of images comprise a plurality oflocations captured under a plurality of illumination conditions.
 22. Themethod of claim 21, wherein the light probe illumination parameterscomprise sun parameters and sky parameters.
 23. The system of claim 22,wherein the learned mapping is based on a sun model for the sunparameters and a sky model for the sky parameters.
 24. The system ofclaim 23, wherein the sun parameters comprise sun position, sun angularvariation, and sun color.
 25. The system of claim 24, wherein the skyparameters comprise sky color and sky angular variation.
 26. The systemof claim 23, wherein the sun model is based on a double exponentialmodel.
 27. The system of claim 23, wherein the sky model is based on oneof a Preetham model, a cosine model, or a von-Mises Fisher model. 28.The system of claim 23, wherein the sun model and the sky model arefitted to the light probe database captured light probes to createfitted models.
 29. The system of claim 23, wherein the computer isfurther configured to: insert a virtual object into the outdoor image;and light the virtual object using the predicted light probe.
 30. Thesystem of claim 25, wherein predicting a light probe for an outdoorimage comprises: predicting the sun position based on a probabilisticmodel; and predicting the sun angular variation, sun colors, sky colors,and sky angular variation based on a regression technique.
 31. Thesystem of claim 30, wherein the sun position is predicted relative tothe camera that took the outdoor image.
 32. The system of claim 30,wherein the regression technique is a kernel regression or a linearregression.
 33. A method of creating a light probe database, comprising:capturing a plurality of images for each of a plurality of locationswith a camera; capturing the light probes for each of the plurality ofimages; and recovering image features for each of the plurality ofcaptured images.
 34. The method of claim 32, wherein recovering imagefeatures for each of the plurality of captured images comprisesrecovering a normal map for each of the plurality of captured images.35. The method of claim 33, wherein the plurality of images comprise aplurality of locations captured under a plurality of illuminationconditions.
 36. The method of claim 34, further comprising calibratingand aligning the captured images and captured light probes.
 37. Themethod of claim 35, wherein the light probe database is used to predicta light probe for an outdoor image.